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Foreword 



Since the early development of quality assurance in higher education, external 
reviews have been part of quality assurance procedures. In its Recommendation 
of 24 September 1998, the Council of the European Union required that “quality 
assurance procedures should generally consist of an internal, self-examination 
component and an external component based on appraisal by external experts”. The 
Council further called for the involvement of all relevant players, including students 
and foreign experts, in reviews and the publication of reports. 

The Standards and Guidelines for Quality Assurance in the European Higher 
Education Area (ESG), adopted by ministers of higher education of the Bologna 
signatory countries in 2005, have as their main results and recommendations not only 
cyclical internal and external evaluations of higher education institutions and/or their 
programmes, but also a periodical review of European quality assurance agencies to be 
undertaken preferably at national level. 

External reviews became a membership requirement when ENQA became an 
association in 2004. They also became a condition of application for inclusion in the 
European Quality Assurance Register for Higher Education (EQAR). Reviews are 
ultimately an integral part of the European quality assurance scene. 

Following the completion of 19 ENQA members’ external reviews, the ENQA 
seminar held in Paris in July 2008 took place at an appropriate time to take stock of the 
achieved reviews to learn lessons from these first outcomes and, hence, continuously 
enhance this process, essential for strengthening mutual trust between agencies and 
stakeholders in higher education as well as among the agencies themselves. 

This report gives an overview of the background and purposes of external reviews, 
and presents the perspectives and experiences of agencies and expert teams on the 
review process. I am confident that this publication will be valuable for and will arouse 
the interest of all quality assurance agencies, especially those that are planning to be 
evaluated. 

Achim Hopbach, 

President 

European Association for Quality Assurance in Higher Education (ENQA) 



Introduction 



Nathalie Costes, Project Manager, ENQA and Bruno Curvale, Head of international affairs, 
AERES 

There are internal and external purposes for which quality assurance (QA) agencies are 
required to go through an external review. Reviews may be carried out with a view to 
enhance the operations of the agency and/or the operation of the national QA system. 
At the external level, external reviews are one of the conditions of membership for 
ENQA member agencies. According to the ENQA Regulations, agencies are required 
to be externally evaluated before applying for the first time for Full membership of 
ENQA. Once they have been granted Full membership, agencies must successfully 
undergo an external review at least once every five years. They must demonstrate that 
they comply with the membership criteria, which consist of part three of the Standards 
and Guidelines for Quality Assurance in the European Higher Education Area (ESG) 

- as adopted by the European Ministers of higher education in Bergen in 2005 - and 
additional requirements and guidelines. In addition to Full membership of ENQA, 
agencies are also required to submit a review report when applying for inclusion in 
the European Quality Assurance Register for Higher Education (EQAR). EQAR was 
proposed at the Bergen ministerial summit in 2005 and agreed at the London summit 
in 2007. This Register of recognised quality assurance agencies operating in higher 
education within Europe was established in order to increase transparency, visibility 
and comparability of quality of agencies. This goal is reached through regular national 
external review by peers and other experts - including publication of the criteria, 
methodologies and outcomes of the review - against the ESG. 

As mentioned above, ENQA membership and EQAR listing are each based on 
substantial compliance with the ESG, to be verified through an external review. This 
explains why the same review is accepted as sufficient for both ENQA membership and 
EQAR listing purposes. In addition, the four founders of the Register (EUA, EURASHE, 
ESU and ENQA) agreed that full ENQA membership will normally constitute 
satisfactory evidence for inclusion in the Register. 

The principle of subsidiarity underpinning the ESG implies that reviews are 
normally initiated and coordinated at national level by national authorities. Where 
this arrangement is not appropriate or possible, reviews maybe organised by ENQA 
or by an international quality assurance organisation. Whether conducted at national 
or international level, external reviews share the same objective, which is to examine 
agencies’ compliance with the ENQA membership criteria/ESG. 

In both cases, ENQA membership and EQAR listing, assessment of the degree of 
compliance with the ESG is the result of a three-step process. First, the agency carries 
out a self-evaluation; then it undergoes an external and independent evaluation; 
and finally the ENQA Board, assisted by its standing Review Committee (and by the 
Register Committee in the near future), assesses the agencies’ applications (i.e. the self 
and external evaluation reports). This process raises many questions, regarding both 
the implementation and quality of the various stages: 

• the self evaluation and the preparation for external evaluation; 

• the independent external evaluation, which involves examination of documents, 
site-visits and interviews; 



• the final assessment of applications, which is primarily based on the conclusions 
of the external evaluation. The Board also carefully considers the quality of the 
evaluation. 

• The use of qualitative criteria and judgements, the implementation of the notion 
of substantial compliance, as well as the specificities of national approaches all 
contrive to make this process even more complicated. 

The present report is the result of the ENQA seminar “First external evaluations of 
quality assurance agencies - lessons learned”, hosted by AERES in Paris on to and n 
July 2008. The seminar was organised after 19 ENQA member agencies had undergone 
an external review, coordinated at national level, by ENQA or by an international 
QA agency. It aimed to provide an opportunity to reflect on the various stages of the 
process from the experience gained by the agencies, the review panel members and 
the members of the ENQA Review Committee. 

The purpose of the seminar was threefold: 

• to draw conclusions from reviews that had already been conducted and, in 
particular, to point out good practices and possible areas for improvement; 

• to raise the awareness of the potential actors of external reviews (managers of 
agencies and experts) on the difficulties encountered during the reviews and 
to reflect specifically on the training for experts; 

• to reflect on the implementation of centrally important concepts or principles 
to increase the quality of the review results. 

The seminar endeavoured to better define two particular concepts that are at the 
core of the external evaluation process: the notion of "substantial compliance", which 
defines the type of approach that the evaluator has to implement; and the notion of 
independence, which assures the legitimacy of the agencies' activities. 

During plenary sessions, an introduction on the background and purposes of 
external reviews was given, as well as a presentation on the differences, similarities 
and challenges between nationally and ENQA coordinated reviews. Review experts 
had an opportunity to relate their experience and to highlight and raise awareness of 
the challenges of conducting self-evaluation and external review from the perspective 
of the agency and the review team. In addition, admission procedures to ENQA and 
to EQAR were discussed. Meta-evaluation of the review results was analysed as an 
essential element in the context of external reviews of agencies. 

During working group sessions, participants were asked to revise and complement 
the two ENQA documents pertaining to reviews: the Guidelines for national reviews 
of ENQA member agencies and the Briefing pack for review panel members of ENQA 
coordinated reviews. The workshops allowed participants to compare their views, 
to pursue the reflection initiated at the Warwick workshop in 2006 and during the 
Quality Convergence Study II Promoting epistemological approaches to quality 
assurance, and finally to make suggestions on how to improve the robustness of 
evaluation procedures. 

The following report includes four articles, submitted by the speakers and based on 
their presentations, which introduces the background and purposes of external reviews, 
as well as the perspectives and experiences of agencies and expert teams on the review 
process. 



Chapter 1: Background and purposes 
of external reviews of Quality 
Assurance (QA) agencies: 

ENQA membership, national purposes 

Tibor Szanto, Secretary General of the Hungarian Accreditation Committee (HAC) 

1. Who will guard the guardians? 

In his classic work on the ideal state, Plato devotes an extensive part to the selection, 
education and training of guardians, the constituents of the state being responsible 
for the protection of the citizens and the defence (and enlargement, when needed) 
of the territory of the state 1 . At one point of the discussion, Socrates, the leading 
character in Plato’s works, says that guardians should refrain from being drunk. He 
explains that this state of mind simply does not fit a guardian’s character and important 
responsibilities. Another discussant, Glaucon, replies to this, stating that it would be 
ridiculous to guard the guardians themselves. 

[Socrates] That they must abstain from intoxication has been already remarked by 
us; for of all persons a guardian should be the last to get drunk and not know where 
in the world he is. 

[Glaucon] Yes, he said; that a guardian should require another guardian to take care 
of him is ridiculous indeed 2 . 

In another translation: 

[Socrates] “From intoxication we said that they must abstain. For a guardian is 
surely the last person in the world to whom it is allowable to get drunk and not 
know where on earth he is.” 

[Glaucon] “Yes,” he said, “it would [be] absurd that a guardian should need a 
guard .” 3 

That is, due to their careful selection and training, guardians, by their own nature and 
virtue do not require further guarding or control whatsoever. 

About five centuries later, another classic author, Juvenal, dealt with this question 
again in his sixth Satire. The poem is about the relationship of men and women, 
about marriage and, more specifically, about the “nature” of women. Fidelity and 
trustworthiness are also discussed among other important issues. Juvenal is sceptical 
about these; the message of the poem in this respect is that women cannot really be 
trusted and, what is important for us now, neither can their guards! (Actually, the 



1 Plato, The Republic. 374e - 412b 

2 ibid. 403e, http://classics.mit.edU/Plato/republic.4.iii.html 

The Dialogues of Plato translated into English with Analyses and Introductions by B. Jowett, M.A. in Five Volumes. 3rd edition 
revised and corrected. Oxford University Press, 1892. 

3 http://www.perseus.tufts.edu/cgi-bin/ptext?lookup=Plat.+Rep.+3.403e 

Plato in Twelve Volumes, Vols. 5 & 6 translated by Paul Shorey (1935). Cambridge, MA, Harvard University Press; London, 
William Heinemann Ltd. 1969. 



question used as the title of this section originates in its widespread form from this 
work of Juvenal and the original reads: Sed quis custodiet ipsos custodes P 4 ) 

I hear always the admonishment of my friends: 

“Bolt her in, constrain her!” But who will guard 

the guardians? The wife plans ahead and begins with them! 5 

Another translation: 

I know well the advice and warnings of my old friends —’’Put on a lock and keep 
your wife indoors.” Yes, but who is to ward the warders? They get paid in kind for 
holding their tongues as to their young lady’s escapades; participation seals their 
lips. The wily wife arranges accordingly and begins with them 6 . 

Thus, according to Juvenal, guards cannot be trusted. 

The question is rather important since it relates not only to marriage or the 
defence of the state, but in general it sheds light to the strong correlation between 
the effectiveness of enforcing moral behaviour, and the morality of the enforcers 
themselves. In political terms, this is the question of where the ultimate power should 
lie. 

But we do not need to enter here into a discussion of the subtleties of ethics and / 
or political science. The question is highly relevant to quality assurance of higher 
education, too. “Guardians” in our case are quality assurance agencies, since they are 
“guarding” the quality of higher education by regularly evaluating the operation and 
standards of higher education institutions (HEIs) and study programmes. In general, 
the question in the title of this section relates in our terms to the trust of HEIs and all 
the other stakeholders in the activities of QA agencies, with an aim at ensuring the 
quality of the QA agencies themselves. 

2. Quality of QA agencies 

Although not necessarily inspired by classical authors, QA agencies have since been 
devoting serious attention to this issue, and beyond implementing internal measures 
and mechanisms, quite many of them have formally demonstrated to their stakeholders 
and the public that they are reliable. These QA agencies have also demonstrated that 
the quality of their own operation meets some externally determined standards, 
within the past ten to fifteen years. In the United States the first QA (accreditation) 
organisations were established in the early 20 th century. The CHEA (Council for Higher 
Education Accreditation ), the umbrella organisation of accreditation agencies in the 
USA, first introduced its recognition scheme in 1998. The essence of this is that CHEA 
officially recognises all those accreditation agencies which meet CHEA’s recognition 
standards. The standards are related to the operation of the agencies and reflect good 
practice and external expectations. Both the standards and the list of recognised 
agencies are public and available on the CHEA website 7 . It must be mentioned that 
in addition to CHEA, the US Department of Education has a recognition scheme for 
accreditation agencies. 



4 http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3F 

5 ibid. 

6 Translation by G.G. Ramsay, http://www.fordham.edu/halsall/ancient/juvenal-satvi.html 

7 www.chea.org 



INQAAHE, the worldwide network of QA agencies, devoted its 1999 biennial 
conference held in Chile to the theme of evaluation of evaluation agencies 8 . 
Presentations and discussions of this conference led to the quality label initiative which 
finally resulted in INQAAHE’s Guidelines of Good Practice, a document describing (in 
its current version) 12 standards for QA agencies to be considered as reference points 
for operation and possible external agency reviews 9 . 

ENQA organised a special workshop in Sitges, Spain, on 27 February - r March 
2003, entitled “Taking our own medicine: How to evaluate quality assurance agencies 
in order to create trust in their work and thereby in higher education” where case 
studies of internal QA solutions of agencies and external evaluations of agencies were 
presented and, based on this, participants discussed various ways and methods of 
enhancing the quality of QA agencies’ operation. In regards to Quality Assurance, 
this workshop served as one of the immediate antecedents for the Berlin ministerial 
meeting held in September. In that same year, ministers entrusted ENQA and the other 
three E4 organisations (EUA, EURASHE and ESIB) to develop the general framework 
of QA in the European Higher Education Area. 

“[. . .] Ministers call upon ENQA through its members, in co-operation with the 
EUA, EURASEIE and ESIB, to develop an agreed set of standards, procedures and 
guidelines on quality assurance, to explore ways of ensuring an adequate peer 
review system for quality assurance and/or accreditation agencies or bodies [. . .]” 10 

This was the mandate that led to the creation of the Standards and Guidelines for 
Quality Assurance in the European Higher Education Area (ESG). The ESG were the 
result of a joint effort of the E4 organisations and, in addition to a core of experts 
directly working on various preparatory materials and drafts of the final document, 
hundreds of professionals representing many countries [higher education institutions 
(HEIs) and QA agencies] contributed in one way or another during the one and a half 
year preparation period. The ESG were adopted by the ministers in Bergen in 2005, 
and they presented not only standards and guidelines related to the internal QA of HEIs 
and the external QA mechanisms, but also standards and guidelines for the operation 
of QA agencies. Moreover, in a separate chapter, they discussed the possible elements 
of a peer review system for QA agencies and proposed the creation of a register of 
trustworthy QA agencies operating in Europe. 

All these initiatives and events served the purpose of ensuring and demonstrating 
that guardians of the quality of HE are doing their job properly and persistently, and 
they can really be trusted. 



8 A framework for Quality: Evaluating Evaluation. Papers presented at the INQAAHE Conference, 2-5 May 1999, Santiago de Chile 

9 Guidelines of Good Practice. INQAAHE, 2003-2007, consecutive editions, http://www.inqaahe.org/main/capacity-building-39/ 
guidelines-of-good-practice-5 

10 “Realising the European Higher Education Area” Communique of the Conference of Ministers responsible for Higher Education 
in Berlin on 19 September 2003. 



3. Evaluations of Agencies 

Parallel to and in cases preceding all the above, agencies in various parts of the world 
took care of their quality and reliability and the demonstration of them to stakeholders 
and the public at large. 

It is well known that according to the ESG (and thereby the ENQA membership 
criteria) quality assurance agencies are expected to undergo an external review every 
five years. Standard 3.8 of the ESG (numbering according to the 2009 edition) relates 
to the accountability procedures of agencies: 

Agencies should have in place procedures for their own accountability 11 . 

Guideline No. 3 under this standard suggests: 

3. A mandatory cyclical external review of the agency’s activities at least once every 
five years. 

There were quite a few agencies, however, which had been reviewed even before 
these expectations were first published in 2005. External evaluations of agencies were 
commissioned and conducted; reports on the results published. To list a few of them 
with the date of the review indicated: 

• MSA, Middle State Association of Schools and Colleges (USA), 1990 

• HEQC, Higher Education Quality Council (UK), 1993 

• NCA, North-Central Association of Schools and Colleges (USA), 1996-98 

• AAU, Academic Audit Unit (New Zealand), 1997 and 2001 

• EVC, Evalierungscenteret (Denmark), 1998 

• OCGS, Ontario Council of Graduate Studies (Canada), 1999 

• UGC, University Grants Committee (Hong-Kong), 1999 

• HAC, Hungarian Accreditation Committee (Hungary), iggg-2000 

• GAC, German Accreditation Council (Akkreditierungsrat) (Germany), 2001 

• EUA, European University Association, 2001-02 

Lessons learnt from the seven 12 cases set in italics above were summarised and 
published in 2005 13 . 

The reasons and goals of the external evaluation were usually as follows: 

a. An evaluation/accreditation cycle ended; 

b. Irrespective of cycles, past activities of the given agency were evaluated; 

c. The evaluation was to serve planning future agency activities; 

d. The evaluation had a (national) system level focus; 

e. It was already a second review (follow-up of previous evaluation); 

f. The evaluation was to demonstrate meeting external expectations. 

This latter goal became more stressed after the publication of the ESG. 

It must be mentioned that in addition to the case of AAU (the New-Zealand agency on 
the list above), several other agencies have already been reviewed for the second time 
since 2005, namely the Danish Evaluation Institute (its current name being EVA), 



11 Standards and Guidelines for Quality Assurance in the European Higher Education Area. Helsinki: ENQA, 2009. p.26. Accessible 
e.g. at http://www.enqa.eu/files/ESG_3edition%20(2).pdf 

12 AAU, Academic Audit Unit (New Zealand) was evaluated on two separate occasions, in 1997 and 2001. 

13 T.R. Szanto, Evaluations of the third kind: External evaluations of external quality assurance agencies” Quality in Higher 
Education 11(2005) 183-193. 



HAC, GAC, and the second review of EUA is also in progress. This means that the ESG 
expectation of the cyclical nature of agency reviews seems to work in practice. 

Moreover, the ESG was a milestone also in the respect that it explicitly defined 
external agency reviews as an internal quality assurance element of the operation 
of agencies. 

4. ENQA membership and national purposes 

After the publication of the ESG, the Board of ENQA decided to deal with the question 
of agency reviews in detail and to give some advice to its member agencies as to how 
exactly such reviews should be conducted and what the role of ENQA in reviewing 
agencies would be. 

From the point of view of ownership, the Board defined two types of reviews: 

a. National review, being the “default”, the one applied normally and in most cases. 

b. ENQA co-ordinated review, being available only where, for good reasons, 
a national review would not be appropriate or possible. 

It must be added that only the ownership and organisation of the review is meant by 
“national”, the review team is (should be) international even in national reviews. And, 
irrespective of the ownership of the review, it should be independent, transparent, and 
robust, that is, it should be objective and provide ample evidence for the given agency’s 
meeting (or not) the ESG and thereby the ENQA membership criteria. 

Further, as to focus, a review can be either of type A, concentrating only on 
compliance with the ESG and thereby on ENQA membership, or type B, having other 
purposes than granting or reconfirming ENQA membership. 

The related ENQA document where detailed information is available is the 
Guidelines for external reviews of quality assurance agencies in the European Higher 
Education Area which includes the former (i) Guidelines for national reviews of ENQA 
member agencies, (ii) Briefing Pack for review panel members of ENQA co-ordinated 
reviews and (iii) Principles for ENQA coordinated reviews. 

The Guidelines for national reviews of ENQA member agencies and the Briefing pack 
for review panel members of ENQA coordinated reviews were discussed during the Paris 
seminar. 

Seen from a historical perspective, as to its general goal, the review can serve the 

a) Agency agenda (enhancement of the operations of the agency) 

b) National, system level agenda (enhancement of the operation of the national 
QA system) 

c) ENQA membership (granting or reconfirmation of membership) 

d) EQAR listing (European Quality Assurance Register ) 14 

Of course, any combination of the above goals is possible and even all four goals 
can be served by one and the same review. So much the more as the granting or 
reconfirmation of ENQA membership based on an external review constitutes prima 
facie evidence for the EQAR listing, article 2 of the EQAR Procedures for applications 
reads: 



14 http://www.eqar.eu/ 



“Full membership of ENQA normally constitutes satisfactory evidence for 
substantial compliance with the ESG” 15 . 

However, it is interesting to note that external reviews of agencies originally served 
internal purposes basically, such as the enhancement of a given agency’s operation or 
a national QA system. External aspects and publicity were secondary only. Whereas 
recently, especially after the publication of the ESG, it seemed that the external 
purpose (ENQA membership, EQAR listing, demonstrating compliance with external 
standards) was more and more prevailing while internal quality enhancement seemed 
to become secondary. It seems that it is more important nowadays to show how good 
one is than to be good, just for one’s own sake, for goodness itself. In other words, as to 
agency reviews, a purpose shift from the internal towards the external can be observed 
historically. 

If we widen the focus and consider the overall higher education QA scheme, it seems 
as if current trends were opposite in relation to HEIs and agencies. Evaluations of HEIs 
seem to focus more recently on quality enhancement than external accountability. 

A sign of this can be that institutional evaluations or audits come to the fore while 
assuring the quality of individual programmes is left to HEIs in a growing number of 
cases 16 , although this is still not the majority 17 . As to agencies, on the other hand, the 
shift from (internal) improvement towards (external) accountability seems to prevail, 
as was discussed above. 

Finally, we might also ask the question: Are standards and guidelines in general, 
or the ESG in particular, more and more used as norms instead of reference points, 
no matter what the original intentions were? 

5. The use of external evaluations: how many guardian levels do we need? 

We are witnessing an interesting process of development of quality scrutiny schemes. 
More than a hundred years ago, there were only higher education institutions taking 
care of their own programmes, standards and quality. This, we can say, is the first level 
of assuring quality. Then, at the beginning of the 20th century, accreditation appeared 
in the United States. It originally emerged as something like self-defence of some 
HEIs against the more and more new providers representing dubious quality. External 
quality assurance of HEIs reached Europe in the eighties and since then it appears that 
all countries in Europe have had external QA agencies of a kind. Thus, on the second 
level, QA agencies scrutinise and take care of the quality of institutions and their 
programmes. But no, this is not the end, agencies themselves must be scrutinised as to 
their own quality and reliability. Agency reviewers appeared on the third level in the last 
couple of decades, though still not really as separate organisations (“meta-agencies”) but 
rather as occasional individual review teams only. Fine. Anything else? Do not worry, 
here is the fourth level, on which professional entities (like “meta-meta agencies”?) 
such as ENQA and the Register Committee (RC deciding on EQAR listing) review the 

15 http://www.eqar.eu/fileadmin/documents/eqar/official/RC_01_l_ProceduresForApplications_vl_0.pdf 

16 See the presentations of the ENQA workshop on “Programme oriented and institutional oriented approaches to quality assurance: 
new developments and mixed approaches”, Berlin, Germany, 12-13 June 2008. 
http://www.enqa.eu/eventitem.lasso?id=165&cont=pasteventDetail 

17 An ENQA survey conducted in 2007-2008 found, among others, the following. „The analysis of the types of activities covered by 
agencies shows that, whilst two-thirds of agencies use programme level procedures, just less than half work at institutional level.” 
N. Costes et al., Quality Procedures in the European Higher Education Area and Beyond - Second ENQA Survey. ENQA 
Occasional papers 14. Helsinki: ENQA, 2008. p. 8. http://www.enqa.eu/files/ENQA%200ccasional%20papers%2014.pdf 



review of the reviewers, that is, they scrutinise whether or not the agency reviews were 
conducted in a proper way and the results of them are reliable. 

Thus, the quality assurance pyramid of higher education - currently, at least - looks 
like this: 

ENQA, RC 
Agency reviewers 
External QA agencies 
Higher Education Institutions 

Have we already reached the top? Or are there some further, even higher levels 
waiting to be established? Additionally, who can assure us that ENQA or the Register 
Committee does a perfect, unquestionable job? Can we really trust them? 

We started with the question: Can external QA agencies be trusted? And we ended 
up asking: Can agency reviewers be trusted? What is more: Can the reviewers of the 
agency reviewers be trusted? 

Who should guard the guardians then? And how many guardian levels do we need? 
What is reflected in the creation of the multiple levels of quality assurance? What is 
this all about? Is it still about higher education or is it about power and politics? Is it 
about something else? What is this multi-layer structure good for? Who are the real 
beneficiaries? Are there any? Yes, I do hope. 

I finished my presentation in the Paris seminar by telling a story about Bertrand 
Russell, the 20th century British philosopher who once visited India. Discussing with 
some colleagues in India, Russell asked them about their worldview or cosmogony, one 
could say. 

- The universe rests on a nice lotus flower - the answer was. 

- And where is that lotus flower? Is it just hanging in the air? - asked Russel. 

- Not at all. Naturally, it is in a nice lake. 

- And how about that lake, where is it? 

- It is on the back of an elephant. 

- OK. And where does that elephant stand? 

- Well, it stands on the back of a turtle. 

- And the turtle? 

- It is on the back of another turtle. But don’t bother Mr. Russel anymore: there 
are turtles, all the way down. . . 



Chapter 2: Self-assessment of agencies 
and self-assessment report: an agency 
perspective - the case of the National 
Agency for Quality Assessment and 
Accreditation of Spain (ANECA) 

Rafael Llavori, Head of Unit for Institutional and International Relations, ANECA 

1. Understanding the European Standards and Guidelines (ESG) 18 

- THE WAY FORWARD 

Before starting the self-assessment process, a clear need to analyse the ESG in the light 
of the Spanish context, taking into consideration both the running agency and the 
higher education system, emerged. It was the first time that ANECA underwent such a 
process - beyond the certifications of internal quality assurance systems - and external 
reviews of organisations are not a common practice in Spain. 

This preliminary analysis meant to understand ANECA’s processes and procedures 
in the particular context of the ESG before getting started on writing the self- 
evaluation report. It basically consisted in “translating” the ESG terminology and 
Weltanschauung (“world conception”) into ANECA’s vocabulary in order to be sure that 
all the actors involved (both panel and staff) spoke the same language and, therefore, 
to avoid misunderstandings. 

To begin with, the Direction along with the co-ordinator of the process decided 
that the analysis should be based on both Parts 2 and 3 of the ESG. The analysis also 
included a short overview of the external evaluation programmes and procedures for 
quality assurance led by ANECA trying to identify to what extent the criteria of the 
different programmes were in correlation with the standards in Part 1 of the ESG, 
devoted to internal quality assurance (QA) within higher education institutions (HEIs). 

This led the review co-ordination team, consisting of three people (one person in 
charge of the self-evaluation report and two assistants), to write a self-evaluation short 
guide explaining each standard under the magnifying glass of the regional and national 
contexts with illustrating examples. To this end, a first identification of evidence related 
to every standard was attempted. The review co-ordination team was then enlarged to 
include nine more people (thus forming the Self-Evaluation Committee) coming from 
both different departments and positions within the agency. In that way, a manifold 
perspective, both vertical and horizontal, was ensured. 

ANECA’s way of interpreting the ESG resulted in a particular picture which had 
probably been the leitmotiv and the silent symbol of the whole process for the agency 
(see Figure 1). It is a visual way of presenting the ESG as a concentric disposition of 
circles: 
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• The inner circle corresponds to Part 1, ESG for internal QA within HEIs 

• The circle in the middle corresponds to Part 2, ESG for the external QA of higher 
education 

• The outer circle corresponds to Part 3, the ESG for external QA agencies 




Figure 1. 

2. Internal review of a quality assurance agency: the case of ANECA 

- COMPILING THE SELF-EVALUATION REPORT 

The process of compiling a self-evaluation report required ANECA to set up a suitable 
and specific team, composed of ANECA staff members from various departments. One 
basic point in the preparation for the review was to make sure the evaluation would 
be perceived as a real strategic objective of the agency as a whole, with a participative 
bottom-up process, and not only as one department’s special task. 

Therefore, a high staff involvement at all levels in the objectives of the review 
was fostered and a clear leadership consisting of the review co-ordination team was 
identified. 

The appointment of this steering group allowed the establishment of a strict 
follow-up of the tasks assigned to the different members of the enlarged Self-Evaluation 
Committee. All of them had to devote part of their working time to this review 
procedure in addition to their department commitments. 

But this working scheme could not have been feasible without a clear implication 
of the Board of directors, the staff of ANECA and the Advisory Board. Therefore, this 
participative process was not for informative purposes only, but was also aiming at 
dissemination activities. 

The review co-ordination team became a reference point for the preparation of the 
self-evaluation report, which took place in the following way: 

• Self-evaluation guide 

• Fixed schedule 

• Periodical meetings of the Self-Evaluation Committee 

• On-going information and training process 

• Evaluation agenda with a clear deadline 

• Collecting and organising the evidence for clear and easy access 

• Translating the documentation into English 

• Writing the self-evaluation report 

• Preparing the site visit 





- DISSEMINATING INFORMATION WITHIN THE AGENCY 

The steering group co-ordinated the spread of information into two levels: to the Self- 
Evaluation Committee (information derived from the process) and then to the rest 
of the staff and to the ANECA Board of Directors at regular meetings (every 15 days) 
where the on-going process of the report was monitored. These periodical meetings, 
where the persons in charge of the Units had to publicly admit whether they fulfilled 
their commitments or not, were ironically called “breakfast at ENQA’s” because they 
were held early in the morning and this name was referring to the well-known movie 
“Breakfast at Tiffany’s”. The famous picture of Audrey Hepburn, who is the icon of the 
film, was attached to the e-mails together with the agenda of the meeting. 

Electronic internal tools such as intranet and the internal newsletter were 
thoroughly used to disseminate information and collect data for the process. 

- THE SELF-EVALUATION REPORT 

Writing the self-evaluation report marked the final step in this collective process. 

The report was structured as follows: 

• Introduction 

- Background 

- Aim of the external evaluation 

- Spanish university system 

- ANECA 

• Self-evaluation of ANECA 

- Background 

- Self-evaluation process 

• Analysis of the ESG 

• Overall assessments 

- Overall assessment 

- Opportunities for improvement 

• Annexes 

- Glossary 

- List of evidence 

- Organisation chart 

The first draft was first circulated to a wider group including the presidents of the 
Committees, the Technical Committee and the Advisory Board before being submitted 
to the members of the Board of Trustees (i.e. the governing body of the Agency). 

3. Lessons learned 

The completion of the self-evaluation procedure while preparing for the external 
review allowed a real staff awareness of the three main aspects we were probably 
only implicitly aware of: (i) learning about our own organisation in a thorough way, 

(ii) creating a new methodology for internal work and sharing information in a real 
horizontal way, (iii) being aware of the weaknesses of ANECA rather than being aware 
of only ‘sort of problems’ raised through the work, but also being well cognisant of the 
strengths of ANECA. 

For these reasons, and as an important result rather than a conclusion, we learned 
that we could not undergo the external review at an earlier stage because both the 



agency and staff would have lacked maturity and the national legal framework 
would have been incomplete. At the same time, undergoing the review at a later date 
was certainly too late for the agency to be prepared for the implementation of the 
accreditation process (which took place immediately after the process) and to have 
the needed international support and recognition from the Spanish stakeholders and 
international organisations. 



Chapter 3: External reviews of quality 
assurance agencies - Perspective of 
the expert team 

Thierry Malan, Former Inspector General of the National Education and Research 
Administration - France 

1. Introduction 

Before and during the review process, external reviewers are strongly recommended 
to keep in mind the following two points, essential for a successful review: 

• consider the context of the agency under review; 

• harmonise their approaches and views, and especially get a common 
understanding of the notion of compliance with the European Standards and 
Guidelines (ESG ) 19 and ENQA membership criteria. 

2. Consideration of contexts 

In the country where it is established, the reviewed agency is inserted in a complex 
network of higher education institutions (HEIs) and processes. It shares tasks together 
with other relevant stakeholders in higher education, especially Ministries, funding 
authorities, and HEIs. This division of tasks is the temporary result of cultural and 
historical influences. The diversity of cultural and historical backgrounds around 
Europe brings about a great variety of legal frameworks, which are sometimes 
very elaborated and regulate the agency’s activities in detail, as well as a specific 
balance between public and private HEIs. Some agencies may be involved, formally 
or informally, in decision-making and funding processes, while others are strictly 
dedicated to quality assessment only. 

All these factors have an influence over the agency’s work, organisation, decision- 
making powers and procedures, and the level of its resources. 

The agency under review is also part of a specific quality assurance institutional 
context. Which are the different evaluation actors and their roles? There are three 
possible situations where there is (are): 

• a single national agency with legal exclusivity; 

• several national public agencies, which may have a specialised or general remit, 
and a national or regional primary domain. Such a situation may entail possible 
duplications, overlaps, competition, coordination and arbitration mechanisms 
between agencies; 

• several agencies, including private, and/or foreign agencies. 

National legislation can assign monopoly rights to quality assurance agencies or 
instigate competition between them: in that case, are HEIs allowed to apply for 
external quality assessment to any agency in the country/region or abroad? 
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Expectations from agencies maybe narrowly specified or very broad. Each of 
their missions, specified by law (evaluation, quality assurance, accreditation), are 
more or less comprehensive, ranging from all aspects of HE evaluation to a single 
defined function, such as programme accreditation, institutional evaluation, research 
institutions and programmes, personnel recruitment and promotion, services, libraries, 
facilities, public policies. 

Appraising the quality of an agency’s activities requires the appraiser to take into 
account the variety of missions and the specific processes and requirements that go 
with their efficient implementation. 

To be eligible for Full membership of ENQA, an agency should have been operating 
for at least two years. 

Agencies have different backgrounds which are reflected in all aspects of their 
activities: 

• some agencies were recently created as a follow-up of the Bologna process; 

• others were founded years ago, and have already seen reforms and significant 
changes within their HE system;. 

• some have replaced former agencies. 

National legislation may require HEIs to be cyclically reviewed (e.g. every three to 
seven years). At the time of the review, some agencies: 

• have evaluated only a few HEIs; 

• are still working partially under former principles and methods of the former 
agency in order to close an evaluation cycle, and are at the same time designing 
and initiating new objectives and sets of rules; 

• have evaluated all HEIs, sometimes even twice or more, which gives elements for 
assessing the improvements of the agency’s processes. 

Other factors play a role in external reviews such as: 

• the variety of provisions for financing and pricing activities, budgets, cost-ceilings 
and diverse resource requirements; 

• the involvement of agencies in public debates, in appeals or courts cases where 
their judgments are called into question; 

• the importance assigned to international recognition (it may happen that 
international recognition, after the external review, is a legal condition for further 
existence of the agency). 

Considering this variety of factors should prevent the review panel from being tempted 
to adopt an inflexible approach when evaluating conformity with the ESG (i.e “rigid 
adherence” to the ESG). 

3. The review process 

The documents produced by ENQA, the Guidelines for external reviews of quality 
assurance agencies in the European Higher Education Area 20 and the review report 
template, provide a clear framework for preparing and conducting the review and site 
visit. 



20 Including the former Guidelines for national reviews of ENQA member agencies, Briefing pack for review panel members of 
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A telephone briefing between the ENQA secretariat, the review panel and an expert 
having participated in other reviews gives a further opportunity to identify and bring in 
possible debatable issues, identify possible problems and prepare for consensus. 

The purpose of the Guidelines is to assist the agency in its preparatory work for the 
self-evaluation report (SER) and the expert team in conducting the external review. 
Further improvement of the Guidelines will probably be possible only through an a 
posteriori comparative analysis by ENQA of past reviews and reports, and their impact 
and consequences. 

Review panels should be composed in such a way that they encompass a variety 
of personal experiences and national backgrounds. It is important that each panel 
includes at least one expert knowledgeable about the HE system, culture and language 
of the country. 

From the agency’s side, the SER is the key piece of the internal assessment process. 
The SER is by itself a valuable indicator, together with: 

• the quality of organisation of its elaboration process; 

• the level of staff involvement in this elaboration; 

• its use as a tool for restructuring priorities of the agency, forwarding important 
issues for the agency and as a learning exercise. 

The SER is expected to be comprehensive, structured according to all relevant ESG 
(parts 2 and 3) and consistent with the evidence provided by the agency. 

It should also be expected to be reasonably self-critical, by pinpointing weaknesses 
and ongoing and foreseen steps for improvements and changes. There is an 
understandable fear of risk of overemphasising weak points and thus creating possible 
misunderstandings. However, there may be more credibility with relatively critical 
comments than with no, or too plain, comments on obvious weaknesses, justifications, 
issues debated and steps taken to cope with identified shortcomings. 

The quality and readability of the website, the availability and readiness to provide 
additional documentation are additional strengths. 

From the review panel’s side, the challenge is to quickly consolidate a team with 
different backgrounds and experiences. 

The site-visit is very short and requires: 

• a good and thorough preparation and assimilation of the documentation provided 
by the agency, and to request, if necessary, additional documents 

• to develop lines of inquiry and an outline report 

• a preliminary meeting of the panel, distribution of roles within the panel, 
provision of short breaks between interviews for internal panel coordination 
during the review. 

For the preparation of the site visit, it is of utmost importance that communication 
between the agency and the panel - through a contact person within the agency and 
the panel’s secretary - is excellent in order to facilitate the identification of major issues 
to be raised and the general organisation and timing of the site visit. 

As for interviews, the language is an important aspect: it has to be clarified before 
the beginning of the site visit meeting in order to give interviewees the possibility to 
express themselves in their own language, as well as in English. 



The draft review report is essentially prepared by the secretary in collaboration 
with the chair of the panel. It is advisable that the first draft is circulated to the panel 
as soon as possible in order to facilitate early inputs from the other team members 
and identification of still debatable points (e.g. level of compliance with the ESG and 
recommendations). 

The final external assessment report is the result of an interactive process within the 
review panel and with the agency (verification of factual accuracy, comments on the 
draft report and conclusions). The principal purpose - the backbone - of the report is 
the assessment of compliance with the ESG. 

The review panel has the option to give a general appraisal of the efficient fulfilment 
by the agency of its missions; this may be an additional opportunity to consider not only 
its present compliance with the ESG, but also its development perspectives. 

4. External reviews of quality assurance agencies are still an emergent and 
learning process 

A great variety of external review reports is to be expected, due to different national 
contexts, levels of dissemination and use of the ESG, composition/backgrounds of 
panels, styles of reports, articulation between findings, consideration of evidence, 
further investigations, assessment of compliance with the ESG, and recommendations 
presented or not as “conditions”. 

For the ENQA Board, this variety of reports raises the issue of ensuring equality of 
treatment when considering the reports and making decisions on membership. How 
does the ENQA Board, and then the EQAR Register Committee 21 , appreciate and 
balance different reports compiled by different review panels? 

This necessary “meta-evaluation” requires consistency of judgements and more 
comparable and homogeneous criteria for membership/inclusion. This is, at the 
present stage, less to be obtained by formal improvements of the Guidelines for 
external reviews than by comparing the first reports as methodological case-studies, 
by improving the debriefing of expert panels (e.g. with a stronger structuring of 
feedback letters to the ENQA Board), and by sharing this gathered experience in 
training workshops. 

The issue of compliance is to be examined in more detail by further discussing and 
specifying the notion of “substantial compliance” on the basis of what has been learned 
from the previous reports. What are the minimum requirements for substantial or full 
compliance with the ESG? Current situations call for differentiating between: 

• sufficient satisfactory compliance (especially for newly established agencies which 
still have to gain experience); 

• if not, identification of steps for progress and follow-up of implementation, 

• beyond the issue of compliance, the pursuit of excellence through further 
continuing improvements and achievements on specific missions and functions. 

Last but not least, ongoing efforts have to be made on terminology issues, since 
the latter can constantly bring confusion during reviews. Possible terminology 
misunderstandings within the panel and between the panel and the agency should 
be resolved as quickly as possible. ENQA should continue to address the challenges 
of communication and language in quality assurance. 
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Chapter 4: Some reflections on 
practicalities and challenges of 
external reviews of QA agencies: the 
perspective of the expert team 

Michael G. Kraft, Austrian Federal Chancellery, Department IV/y: OECD, Science, 
Research, Technology and Education 22 

1. Introduction 

The 1990s have seen a growing state interest in quality of the outcomes of higher 
education and accountability of higher education institutions (HEIs), which led to 
the establishment of national quality assurance agencies. Newton (2007, p. 14), for 
example, stated that “by the end of the 1990s concern for quality and standards was 
global”. This concern went hand-in-hand with reforms in higher education which 
should take better account of students’ and stakeholders’ demands. 

However, some had the impression that it looked more like a neo-liberal approach 
to reform the state, incorporating specifications which seem to minimise the role of 
trust and professional judgement. For example Trow (1994, cited in Newton 2007, p. 
14) spotted a “withdrawal of trust” and Barnett (2003, p. 90) even identified quality 
- like competition and entrepreneurialism - as an ideology which was being levered 
into higher education primarily by the state. At the turn of the century, nobody could 
seriously escape the race for quality in higher education and the proper mechanisms 
seemed to be at hand with the faith that 

“it was assumed that the greater specification of criteria would not only mean a 
more accessible and fairer system for learners but that the trust and interpretive 
judgement that had been a core feature of traditional systems would become 
increasingly irrelevant.” (Young, 2007, S. 454) 

It seemed that state advocates, eager in their will to reform higher education - which 
in itself became increasingly associated with global competitiveness of nation states, 
when lifelong-learning, learning outcome orientation and employability moved to the 
forefront of educational politics - were keen to promote formal and rigid mechanisms 
to reform universities. 

Although “quality has become a project” (Barnett, 2003, p. 90), one should keep in 
mind that “the family of concepts of quality differs over by whom or by which criteria 
that ‘doing something well’ is to be judged” (Barnett, 2003, p. 91). Therefore, before 
applying any kind of quality assurance mechanisms, one has to be clear about what 
aims one is pursuing, since quality is not neutral. 
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2. Quality control vs. quality enhancement 

There has recently been a shift in debates about quality assurance, placing the focus 
on mutual trust and quality enhancement. This shift was also fuelled by the Bologna 
Process, which assigned the prime responsibility for quality assurance to the HEIs. 

For example, the Standards and Guidelines for Quality Assurance in the European Higher 
Education Area (ESG) state that: 

“As their starting point, the standards and guidelines endorse the spirit of the July 
2003 Graz Declaration of the European University Association (EUA) which states 
that ‘the purpose of a European dimension to quality assurance is to promote 
mutual trust and improve transparency while respecting the diversity of national 
contexts and subject areas’.” (ENQA, 2009, p. 12) 

As one can see, a commonly shared objective by the external quality assurance 
community nowadays is to build trust among the different actors in the system. 

The widespread view among external quality assurance practitioners is that it is 
more important to establish an “improvement atmosphere rather than a control 
environment”. On the other hand, national government legislation is moving towards 
accountability, fostering a “checkbox mentality” (e.g. the establishment of the European 
Quality Assurance Register for Higher Education (EQAR) is rather to be interpreted 
in such away). Barnett even argues that “quality, as a state-backed project, is always 
likely to produce a compliance culture” which implies that “a conspiracy in favour of 
compliance is born” (Barnett, 2003, p. 93). This raises the question whether external 
quality assurance agencies are caught between these competing demands of control 
and improvement and whether there is any possible solution to it. How should they 
cope with this fact when organising their procedures for external evaluation? If one 
keeps in mind that 

f. . .] over-specification [...] can lead only to the trivialisation of outcomes and 
the lowering of standards. Trust, we must conclude, is a necessary component 
of any form of assessment and can only be developed over time through regular 
collaboration between peers” (Young, 2007, p. 454) 

It seems reasonable to shift the focus towards quality enhancement. On the contrary, 
also quality assurance agencies themselves have to undergo an external review process 
in order to become member or prolong their membership with ENQA or to be listed 
in the EQAR. This development is more likely to stem from the accountability side of 
quality assurance. Although at first sight this seems reasonably fair, it is nevertheless 
crucial to reflect on this matter when participating in a peer review of quality assurance 
agencies. If the process is only seen as a “check-box mechanism” the procedure will 
be of no use. Albeit there has been some criticism that ENQA, an association of 
quality assurance agencies, decides itself on the (re-)confirmation of membership, it 
is important to see the review process just as one part of the wider ENQA activities, 
which primarily aim at sharing experiences in quality assurance, providing a forum for 
exchange of professional knowledge and finally promoting mutual understanding and 
trust among quality assurance agencies across Europe. 



If we concede that there is no universal, single best framework of external quality 
assurance and that there is always room for improving one’s own system, one will 
have to appreciate the differences between national education systems and policies, 
which are reflected in the workings of quality assurance agencies across Europe. 

When adopting an approach that puts too much strain on and gives priority to formal 
procedures and standards, we are in danger of gradually minimising the role of trust 
and professional judgement 23 . Without doubt, this represents a difficult and sensitive 
undertaking, but it lives up to the different cultural environments and national contexts 
in which different types of HEIs and quality assurance agencies operate. The aim can 
only be plurality in a common but diversified European Higher Education Area (EHEA) 
and not standardisation. There is definitely no need for more of the same. 

3. A hands-on approach - some practicalities for external reviews 

Since there are a number of challenges and pitfalls of external reviews of agencies, 

I would like to document and reflect on some of my experiences of the external review 
of AQU Catalunya in June 2007. Regarding some practicalities for external reviews 
I would like to highlight that it is particularly important that the terms of reference 
state clearly the type of the review (e.g. sole vs. multiple purpose reviews) and therefore 
the assignment of the review team. Given usually a tight timetable, the focus should be 
on the ESG and when additional areas for evaluation are being asked for, it should be 
borne in mind that additional resources and a certain composition of the review team 
might be required (e.g. in case of the teaching staff and research assessment of Spanish 
agencies). 

In order to strengthen the quality enhancement aspect of the review process, 
the agency under review should adopt an approach guided by the principles of 
self-criticism, objectiveness and openness, and should draw up a transparent, 
comprehensible and comprehensive self-evaluation document. It is undoubtedly 
important that the agency conceives the whole process as a chance and challenge for 
its own development and quality improvement and not so much as fulfilling external 
requirements. Only then is quality culture likely to become an internal agenda, 
incorporating a multitude of approaches. 

Concerning the self-documentation, it is also of importance that, although 
a comprehensive list of evidence seems to be very useful, the agency carries out 
a prioritisation of the documents. It should communicate to the review team which 
documents it considers particularly important as evidence to support the findings in 
the self-evaluation report. The latter should also provide the review team with a good 
and comprehensible overview on the higher education system in which the agency 
under review operates in order to be able to appreciate national peculiarities. Moreover, 
one should distinguish between aspects that are under the control of the agency and 
those resulting from national legislation. These different aspects should be clearly 
distinguished in the review report when the conclusions are drawn by the review team. 
When drafting the review report it is important that the conclusions of the review team 
distinguish between the different areas of evaluation on the one hand, but also take the 
consistency of the whole report into consideration on the other hand. 

Although every review is intended to examine as to whether the requirements for 
membership laid down in the ESG are being met at European level, one should also 
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keep in mind that the ESG operate in a context characterised by a strained relationship 
between European Standards and national tradition, context and legislation. On the 
one hand, the ESG are designed to be applicable to all quality assurance agencies in 
Europe, irrespective of their structure, function and size, and the national system in 
which they operate. On the other hand, “it gives only little meaning to ask an agency to 
comply with the ESG if its national legislation distributes roles in the quality assurance 
system in such a way that the agency cannot operate in line with the European 
requirements” (ENQA/NOQA, 2006, p. 14). Keeping this in mind, acceptance of the 
ESG is likely to be broader and it can serve as a solid basis for advancing the EHEA 
across the different higher education communities. 

While the purpose of a European dimension to quality assurance is to promote 
mutual trust and improve transparency, as their starting point the ESG also intend to 
respect the diversity of national contexts. In line with this, it should be acknowledged 
that the ESG recognise the primacy of national systems of higher education and 
the importance of agencies’ autonomy within those national systems. The diversity 
of the EHEA in terms of political and higher education systems, socio-cultural and 
educational traditions, languages, aspirations and expectations, makes a single 
monolithic approach to quality, standards and quality assurance in higher education 
inappropriate. Hence, respecting this diversity and variety implies that a narrow, 
prescriptive and highly formulated approach to standards is inadequate. 

Being aware of this strained relationship between the ESG and national contexts, 
it would be helpful if the ENQA Board could provide the review team members 
with a notion of what substantial compliance means and also communicate this in 
a transparent and comprehensible way to quality assurance agencies and the public. 
Nevertheless, I am quite aware that a certain scope for interpretation will always 
remain. 

With regard to future external reviews for ENQA membership, the experiences and 
findings of former review teams should be made available to other expert panels and 
be taken into account when revising the ENQA documentation on external reviews. 

As already stated above, it should be kept in mind that, although it is about ENQA 
membership, the whole undertaking should be aimed at improvement in character, 
and not control, and guided by the underlying principle of trust. Thus, when it comes 
to interpreting the ESG, substantial compliance as opposed to rigid adherence has to 
be considered as the aim of the whole task. It has to be conceived as an ongoing and 
open process which takes into the focus the discursive nature of the whole undertaking. 
Thus, it is of pivotal importance to assure that future review team members receive 
some form of common training and briefing and that ENQA is willing to constantly 
work on and improve the whole process in order to establish mutual understanding 
and trust among agencies. Although some might consider the present situation as a 
self-referential process (i.e. an association of agencies decides “itself” on membership), 
we should not lose sight of the quality enhancement aspect of the review and the 
importance of building up mutual trust by providing, among other things, a forum for 
exchange of ideas and experiences in order to learn from each other 24 . Nevertheless, 
ENQA activities could be enriched by inviting critical speakers from the scientific 
community who are willing to take up a critical stance in the whole discourse as they 
have a more distant position from practitioners in external quality assurance. Such 

24 For some reflections on the methodology of comparative analyses and mutual observations in order to build up trust among 

quality assurance agencies, see Kraft, 2008. 



seminars could fuel enriching dialogues between practitioners and scientists. As the 
whole endeavour has to be seen in the light of the idea that “the family of concepts 
of quality differs over by whom or by which criteria that ‘doing something well’ is to 
be judged” (Barnett, 2003, p. 91), establishing forums for reflection on methodology 
and fundamental principles might also be useful. In my opinion, any review which is 
solely aimed at inclusion in a register, which has set as one of its aims to allow “quality 
assurance agencies to demonstrate their reliability and accountability at European 
level”, is likely to foster a “checkbox mentality”. 

4. Concluding remarks 

As quality culture has become the new keyword, one should remember that higher 
education policies are permeated by several ideologies. It is in no way clear which one 
will take the lead in the future. Hence, quality, even if seen from its virtuousness, 
will have “to fight its corner” (Barnett, 2003, p. 98). This gets even more complicated 
when we conceive quality assurance agencies as somehow positioned in between the 
demands of national governments and HEIs, since changes in policies also affect their 
workings. 

In my personal view the biggest challenge is to link accountability to quality culture, 
but this can hardly be achieved if quality is used in an opaque and ideological way. It is 
useless and void of meaning to talk about quality and carry out peer reviews if we are 
not clear about what exactly we are doing when making judgements, and to highlight 
our implicit understanding and values which underlie the review process. There is no 
way apart from being critical and willing to constantly evaluate one’s own concepts as 
well as reflect on one’s own understanding of quality and quality assurance. We have to 
make our implicit dimensions explicit and may then move forward, as Christian Thune 
already put it in 2005 in his foreword to the ESG, 

“the long and possibly arduous route to the establishment of a widely shared set of 
underpinning values, expectations and good practice in relation to quality and its 
assurance, by institutions and agencies across the European Higher Education Area 
(EHEA).” (ENQA, 2009, p. 5) 
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Conclusion 



Nathalie Costes, Project Manager, ENQA 

External reviews of agencies are an emerging and learning process to which many 
improvements can still be made. The ENQA seminar was organised in order to learn 
lessons from past reviews and improve the review process. It resulted in the present 
report which contains advice to agencies, future review panels and ENQA. One of 
the recommendations addressed to agencies and review panels is the importance of 
performing a preliminary analysis of parts II and III of the ESG. The review keywords 
in the light of the agency’s national context, historical background, and the national 
higher education system in which the agency operates should also be analysed 
beforehand. In addition to these contextual aspects, all other various factors that may 
affect the review should also be taken into account, in order to determine the elements 
that are under the control of the agency and those that are beyond its control (for 
example, those resulting from national legislation). Such examination should avoid any 
misunderstandings during the process and prompt review panels not to interpret the 
ESG literally, but to examine whether the principles and spirit of the ESG is followed in 
practice. 

Gaining a common understanding of the ESG among the self-evaluation and 
review panels, and more particularly of the notion of compliance with the ESG, was 
seen as a crucial element for a successful review. However, further guidance is still 
needed on the notion of substantial compliance. Participants at the seminar pointed 
out that the indication of a threshold (i.e. the minimum requirements for substantial 
compliance) would be useful for review teams. It was also commonly accepted 
that ENQA and EQAR should work together to consider a joint terminology and 
interpretation on substantial compliance. 

The two keywords of this report are communication and trust. Both self- 
evaluation and final review reports are the result of an interactive process, in which 
communication plays a fundamental role. Good communication and staff involvement 
at all levels of the agency, both vertical and horizontal, are indicators of a successful 
internal evaluation process. They are valuable to disseminate information and collect 
data, and to make sure that the review is perceived as a strategic internal objective of 
the whole agency. The self-evaluation report may also be used to point towards future 
strategies of the agency. Before and during the site visit, excellent communication 
between the agency and the panel is necessary to ensure a smooth and efficient visit. 
Efforts still have to be made on communication and language issues by all players in 
reviews, including ENQA. Trust is now in the spotlight within the QA community. It 
is a necessary component of any assessment. One of the aims of external reviews is to 
build trust among the different actors of the higher education system. In this context, 
it is about trust of HEIs and other stakeholders in QA agencies, and trust within QA 
agencies themselves. The seminar raised questions around the limits of trust and 
the current QA pyramid of higher education with its four levels: HEIs, external QA 
agencies, agency reviewers, and ENQA and EQAR at the top. The meta-evaluation 
process requires trust in the review panel and the evidence provided in the report, 



as well as in the “meta-meta controllers”, namely the ENQA Board and EQAR Register 
Committee. It also requires consistency of judgements. 

External reviews usually have internal and/or external purposes. It is worth noting 
that the importance attached to both purposes has changed over the past years. While 
evaluations of HEIs are now focusing more on quality enhancement than external 
accountability, the contrary seems to happen for evaluations of QA agencies, which are 
experiencing a shift from internal improvement to external accountability purpose. 

A significant move towards accountability and compliance culture may give cause for 
concern. In order to be beneficial to agencies, external reviews should primarily aim 
to foster internal quality improvement. Review panels should always keep in mind 
that, although the review is carried out for ENQA membership and/or EQAR listing 
purposes, it should mainly aim at improvement and not control, and be guided by trust. 
To be credible, QA agencies are expected to be self-critical, objective and open-minded. 
They should conceive the whole process not as one department’s tasks and a fulfilling 
standards exercise, but as an objective of the whole agency to enhance its quality and 
development. 



Annex - Programme of the seminar 



ENQA Seminar 

"First external evaluations of quality assurance agencies - lessons learned" 

10-11 JULY 2008, PARIS, FRANCE 

Organised in cooperation with the French Evaluation Agency for Research and Higher 
Education (AERES) 

Venue: AERES, 20 rue Vivienne, Paris 

PROGRAMME 

Thursday, 10 July 

09:00 Registration 

09:30 Opening address by Jean-Franyois Dhainaut, President of AERES 

09:45 General introduction to the seminar by Bruno Curvale, AERES 

09:55 Evaluation of agencies: the challenges 

Background and purposes of external reviews of quality assurance 
agencies: ENQA membership, national purposes, 

Tibor Szanto, HAC 

ENQA membership and EQAR listing - what repercussions for agencies? 

Emmi Helle, ENQA; Colin Tuck and Lucien Bollaert, EQAR 

ENQA coordinated reviews and nationally coordinated reviews: 
similarities, differences and challenges 

Peter Williams, QAA 

Chair: Guy Aelterman, NVAO 
11:00 Coffee break 



11:30 Panel discussion: Practicalities and challenges of self and external reviews 
Self-assessment of agencies and self-assessment report 

- an agency perspective, 

Rafael Llavori, ANECA ( ENQA-coordinated review in 2007) 

- perspective of the expert team, 

Michael Kraft, Secretary oftheAQU review 

Chair: Bruno Curvale, AERES 

12:15 External review of agencies 

- an agency perspective, 

Seamus Puirseil, HETAC (nationally coordinated review in 2006) 

- perspective of the expert team, 

Thierry Malan, Chair of the AQU review 

Chair: Emmi Helle, ENQA 

13:00 Lunch 

14:00 2 parallel working groups 

Each group, chaired by a facilitator (Rafael Llavori, Thierry Malan), 
will discuss both of the following topics: 

How to prepare for an external review? An agency perspective 

How to prepare for an external review? Perspective of the external reviewers 

15:15 Coffee break 

15:45 Plenary session: de-briefing from the previous workshop sessions 
(elements for the quality of the external reviews of agencies), 
followed by discussion 

Thierry Malan, Rafael Llavori 

Chair: Fiona Crozier, QAA 
17:00 End of the first day 
18:45 Dinner at restaurant Chartier 



Friday, 11 July 



09:00 Recognition procedures of agencies. Meta-evaluation of the external 
reviews: how to assess the quality of external evaluation of agencies? 

Bruno Curvale, AERES 

09:45 3 parallel working groups 

Each group, chaired by a facilitator ( Bruno Curvale, Tine Holm, Guy Aelterman ), 
will discuss the following three topics: 

Meta-evaluation: objectives, role, constraints and difficulties 
Reviewing and complementing the Guidelines for national reviews ofENQA 
member agencies 

Reviewing and complementing the Briefing pack for review panel members of 
ENQA coordinated reviews 

11:15 Coffee break 

11:45 De-briefing from the previous workshop sessions (elements for the quality 
of the recognition processes) 

Bruno Curvale, Tine Holm, Guy Aelterman 

Chair: Achim Hopbach, German Accreditation Council 

12:15 Conclusions from the seminar 

What should be the next steps? Training of external review team experts, 
improvement of the ENQA guidelines for external reviews of agencies? 

Fiona Crozier, QAA 

Chair: Bruno Curvale, AERES 

13:00 Lunch 




The present report is a product of an ENQA seminar on the first external 
evaluations of quality assurance agencies, held in Paris in July 2008. The 
seminar took stock of the achieved reviews of agencies to learn lessons from 
these first outcomes and, hence, provided a platform for continuous 
enhancement of this process in the future. This report gives a background 
and overview of external reviews, and presents the perspectives and 
experiences of agencies and expert teams on the review process. 




